Foundations of Multimodal Representations: A Taxonomy of Representational Modalities

Advances in information technologies are producing a very large number of possible interface modality combinations which are potentially useful for the expression and exchange of information in humancomputer interaction. However, a principled basis for analysing arbitrary input/output modality types and combinations as to their capabilities of information representation and exchange is still lacking. The paper presents an generative approach to the analysis of output modality types and their combinations and takes some steps towards its implementation, departing from a taxonomy of generic unimodal modalities of representation. A small number of key properties appear sufficient for creating a taxonomy of generic output modalities which is both relatively simple, robust, intuitively plausible and reasonably complete. These (orthogonal) properties are: analogue and non-analogue representations; arbitrary and non-arbitrary representations; static and dynamic representations; linguistic and non-linguistic representations; different media of representation; and modality structure. The work presented is being located within the larger research agenda of modality theory.